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Abstract Body 


Background / Context: 

School accountability systems are a popular approach to improving education outcomes in the 
United States. These systems intend to “hold schools accountable” by assessing school 
performance on specific metrics, publishing accountability reports, and some combination of 
rewarding and sanctioning schools based on performance. Additionally, most states and many 
districts rank or categorize schools in some capacity. New York City (NYC), in particular, was 
an early adopter of a letter-grade (A-F) ranking system. School letter grades are intended to 
provide “new” information about school quality and to affect how community members view 
their schools. Ultimately, however, whether accountability letter grades influence stakeholders’ 
perceptions of schools is an empirical question. 

The relationship between accountability and stakeholders’ perceptions of their schools is 
important to understand, because accountability systems are meant to provide new information to 
help parents and students choose better schools. On the one hand, if accountability changes 
parent and/or student perceptions of their schools, policymakers can be more confident that 
accountability does in fact provide useful information to these stakeholders. On the other hand, 
evidence that accountability systems do not affect stakeholders’ perceptions would suggest it is 
unlikely that accountability “works” by helping families make better decisions about schools. 

A large body of research documents the effects of school accountability systems, though few 
studies consider the effects on stakeholder perceptions and no previous research explores effects 
on perceptual congruence within schools. NYC’s letter grade system has been the focus of two 
studies that use a regression-discontinuity approach and find that student achievement increases 
after schools receive “F” letter grades (Rockoff & Turner, 2010; Winters & Cowen, 2012). 
Additionally, Rockoff and Turner examine student, teacher, and parent survey responses in NYC 
and find that after a school receives an F grade, parent evaluations of schools improve, while 
student satisfaction drops; teacher satisfaction is not significantly affected. No studies, however, 
have estimated the effect of school accountability as a system (as opposed to receipt of an “F” 
grade) on stakeholder perceptions or on agreement in stakeholder perceptions within schools. 

This study builds on prior work about “multi-source feedback,” which explores the relative 
perceptions of different individuals, often in the same company. This approach is applied to 
schools in studies of principal effectiveness (e.g., Goldring et al., 2009; Urick, 2012). Few 
studies, however, explore perceptual congruence about the school environment (e.g., Griffith, 
1999, 2000; Urick & Bowers, 2014). This study focuses on between-group congruence (e.g., the 
difference in average parent and teacher perceptions within schools) and within-group 
congruence (e.g., variation in student perceptions within schools). 

Purpose / Objective / Research Question / Focus of Study: 

This paper explores the extent to which NYC’s letter- grade accountability system affects parent, 
teacher, and student perceptions of their schools; in particular, it focuses on agreement about 
academic expectations and discipline, as much research documents the relationship between 
these school factors and student outcomes (e.g., Abdulkadiroglu et al., 2011; Angrist et al., 2010; 
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2011, 2013; Dobbie & Fryer, 2011, 2013a, 2013b; Fryer, 2014; Goddard, Sweetland, & Hoy, 
2000; Phillips, 1997; Shouse, 1996). 


The analysis addresses two key research questions: First, does NYC’s accountability system 
affect parent, teacher, or student perceptions of their schools? Does it improve or worsen 
perceptions, and how does this vary by stakeholder group and school context? Second, does 
NYC’s accountability system affect congruence in stakeholder perceptions, both within and 
between groups? That is, does accountability contribute to a shared understanding of the school 
environment among parents, teachers, and students? 

Setting: 

This study uses New York City data for the 2006-07 (2007) to 201 1-12 (2012) academic years. 

Population / Participants / Subjects: 

The school-level analytic sample includes all middle schools with at least ten percent of parents, 
teachers, and students responding to the Learning Environment Survey (LES). The analytic 
sample includes approximately 95% of all NYC middle schools (please insert Figure 1 here). 
Middle schools are an especially desirable level of schooling for this study, as parents, teachers, 
and students in middle schools all respond to the LES, allowing me to compare parent, teacher, 
and student perceptions. 1 Additionally, there are relatively high survey response rates in middle 
schools. In 2012, for example, 89.6% of students, 82.8% of teachers, and 54.6% of parents in 
middle schools responded to the LES, compared to 76.4%, 85.0%, and 35.0% (respectively) for 
high schools (please insert Table 1 here). The final school-level sample includes 2,127 total 
school-year observations from 2007-2012. The individual-level panel includes all parents, 
teachers, and students in middle schools, for a total of 1,535,084 individual-year observations 
from 2007-2012. 2 

Intervention / Program / Practice: 

New York City introduced progress reports for schools in the 2006-07 school year; specific 
schools were graded for the first time in different academic years, as new schools and those 
without relevant data (e.g., elementary schools without student test scores, high schools without 
graduation rates) are not graded (please insert Table 2 here). This means accountability was 
rolled out across schools within NYC over time. The NYC progress reports present considerable 
information about schools from a variety of sources. The most salient feature of the reports is a 
letter grade (A-F), which represents the NYCDOE’s overall assessment of the school’s 
performance. Overall grades are largely based on students’ academic growth (60 points), 
students’ academic performance (25 points), and the school environment (15 points), though the 
system is complicated and there are changes over time. Integral to the school progress reports is 
the annual LES, given to all teachers, all parents, and students in grades six through twelve 
beginning in the spring of 2007. Importantly, there are several domains (including academic 
expectations and discipline) for which parents, teachers, and students answer nearly identical 
questions (please insert Table 3 here). 


1 Only students in grades 6-12 respond to the LES. 

2 In 2012, for example, there are 187,392 students, 105,920 parents, and 12,783 teacher respondents in the dataset. 
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Research Design: 

I use a difference-in-differences approach to estimate the effect of accountability on stakeholder 
perceptions and perceptual congruence. First, I estimate the impact of accountability on within- 
group congruence using a school-level model, as follows: 

(1) CONGRUENCE s t = po + piACCT st + Sst'p 2 + X st 'p3 + T s t'p4 + a s + 1 + e* 

where CONGURENCE reflects within-group congruence for parents, teachers, or students, and 
ACCT is an indicator variable that takes a value of one if school 5 was held accountable in year t 
and zero otherwise. A school is held accountable in year t if it receives a letter grade in the fall of 
year t (reflecting performance in year t-1). Measuring accountability in this way reflects the 
rollout of letter- grade accountability over time, and Pi will reflect the effect of being held 
accountable (being assigned a letter grade), not the effect of receiving a specific letter grade. The 
model also includes vectors of time-varying school characteristics (e.g., size, attendance rate, 
suspension rate), student characteristics (e.g., race, poverty, educational needs), and teacher 
characteristics (e.g., experience, education) - all measured at the school level. Finally, the model 
includes school effects (a s ) and a linear time trend (t). 

In order to explore between-group congruence, I use a pooled individual-level model, as follows: 
(4) PERCEPTIONS po+piPi+p2Sti + p 3 ACCT s t*T + p 4 ACCTst*Pi + p 5 ACCT st *Sti + S s t'p6 

+ Xst'P? + Tst'P8 + (Xs + t + Sist 

In this model, PERCEPTION is a dichotomous variable taking a value of one if individual i 
(either a parent, teacher, or student) has a favorable survey response. P and St are indicator 
variables reflecting whether individual i is a parent or student (with teachers the reference 
group); thus, Pi and P 2 reflect pre-accountability differences in parent-teacher and student-teacher 
perceptions, and these coefficients can be compared to calculate parent- student congruence. The 
interactions between accountability and respondent type indicators estimate the effect of 
accountability on parent, teacher, and student perceptions. For example, the coefficient on 
ACCT*T reflects the impact of accountability on a teacher’s likelihood of giving a favorable 
survey response. Finally, by adding the coefficients from the individual indicators and the 
individual-times-accountability interactions, I can calculate the post-accountability differences in 
stakeholder perceptions within schools. For example, the post-accountability parent-teacher 
“gap” will be captured by (P + ACCT*P) - (ACCT*T). The difference in pre- and post- 
accountability gaps will reflect the impact of accountability on between-group congruence, and I 
use F-tests to determine if these differences are significant. Thus, results from this specification 
will provide information both about how accountability affects individual perceptions and about 
how accountability affects between-group congruence. 

Data Collection and Analysis: 

I merge data from the NYC progress reports, the LES, and the New York State Report Cards 
(SRCs) to create a longitudinal dataset of NYC schools and students from 2007-2012. Progress 
reports provide letter grades and response rates, the LES provides survey responses, and SRCs 
provide data on school characteristics, such as size/enrollment, teacher turnover, student 
attendance, suspension rates, average student characteristics, and average teacher characteristics 
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Questions on the LES have Likert scale-type responses, for example with individuals answering 
that they strongly agree, agree, disagree, or strongly disagree with a given statement. I first re- 
code all individual responses so higher values reflect more favorable views, and then I create 
separate dichotomous measures that reflect favorable perceptions (a score of 3 or 4 out of four on 
the 1-4 measures). Average stakeholder perceptions (1-4) are relatively high, and parents tend to 
have more favorable views than other stakeholders (please insert Figure 2 here). 

To assess perceptual congruence within schools, I use two different types of measures: within - 
group congruence and between-group congruence. Within-group congruence is measured using 
the coefficient of variation (standard deviation divided by the mean) for each stakeholder group 
within a school. Thus, the coefficient of variation will always be positive; higher values reflect 
greater variation in stakeholder perceptions, relative to the mean, and indicate less congruence. 
Though there is variation across domains, students tend to demonstrate higher within-group 
congruence than parents or teachers (please insert figure 3 here). 

For descriptive analyses, I measure between-group congruence as the difference in average 
stakeholder views within schools (please insert figure 4 here), but but there are considerable 
difficulties with modeling between-group congruence using school-level measures (Cronbach & 
Gleser, 1953; Edwards, 1994, 1995, 2001; Edwards & Parry, 1993; Johns, 1981). I avoid these 
issue by using an individual-level model to estimate the effect of accountability on between- 
group congruence, as previously described. 

Findings / Results: 

Results indicate that in NYC, accountability increases within-group congruence for parents and 
teachers but decreases congruence for students (please insert Table 4 here). Furthermore, 
accountability positively affects stakeholder perceptions of expectations and discipline, with the 
exception that accountability contributes to lower student perceptions of expectations (please 
insert Table 5 here). Differences in the effect of accountability on parent, teacher, and student 
perceptions result in changes in between-group congruence, in most cases decreasing congruence 
by widening existing gaps between stakeholder groups. For example, parents tend to have more 
favorable views of expectations than students in the same schools, and accountability increases 
parent perceptions but worsens student perceptions. 

Conclusions: 

This study will make important contributions to the literature on school accountability by 
providing the first rigorous estimates of how a school accountability system affects parent, 
teacher, and student perceptions of schools, and in particular how accountability affects the 
congruence of stakeholder perceptions within schools. Publicizing new, salient information about 
school quality is a key function of school accountability systems, and this study will provide 
evidence about whether and to what extent school accountability in NYC affects stakeholder 
perceptions. This study will also advance the research on perceptual congruence in schools. Most 
studies of stakeholder congruence in schools use small samples and cross-sectional data to 
explore within- or between-group congruence. This study uses a large-scale survey of parent, 
teacher, and student perceptions to explore congruence within and between three different 
stakeholder groups over time. Further, the large sample size and six-year panel allow for more 
rigorous methods and a more comprehensive analysis of congruence across multiple domains. 
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Appendix B. Tables and Figures 


Figure 1: Analytic sample includes all middle schools with at least 10% of parents, teachers, and 
students responding; 2007-2012 
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Note: Percent of middle schools in analytic sample: 2007 (93%), 2008 (96%), 2009 (98%), 2010 (96%), 2011 
(96%), 2012 (97%) 


Figure 2: Average stakeholder perceptions, 1-4, with 4 reflecting more favorable views, middle 
schools, 2007 
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Note: Bars show the average perceptions of parents, teachers, and students in middle schools across all schools (1- 
4). 
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Figure 3: Within-group congruence - average coefficient of variation in stakeholder views within 
schools, middle school analytic sample, 2007 
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Note: The coefficient of variation is the standard deviation by the mean; thus, a value of “0” indicates complete 
congruence and higher values reflect less congruence. 


Figure4: Average between-group congruence (difference in average stakeholder views within 
schools), middle schools, 2007 
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Note: Bars show the average difference in average stakeholder perceptions within schools, using the 4-point scale. 
For example, a school’s parent-teacher difference for expectations is the school’s average parent perception of 
expectations (1-4) minus the school’s average teacher perception of expectations (1-4). Therefore these differences 
could range from -3 to +3. This figure shows the average of between-group differences across all schools. 
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Table 1: Parent, teacher, and student response rates to the NYC Learning Environment Survey, 
by school type, 2007-2012 




2007 

2008 

2009 

2010 

2011 

2012 

Elementary (%) 

Parent 

35.5 

57.5 

62.8 

66.1 

68.4 

69.4 


Teacher 

49.2 

67.4 

77.3 

78.3 

83.4 

83.1 


Student* 

81.7 

87.9 

93.8 

95.7 

94.5 

94.6 

Middle (%) 

Parent 

21.7 

36.4 

43.3 

50.1 

52.9 

54.6 


Teacher 

42.7 

61.2 

74.3 

77.5 

82.2 

82.8 


Student 

69.8 

84.8 

86.0 

88.7 

89.0 

89.6 

K-8 (%) 

Parent 

30.1 

46.3 

53.0 

56.8 

60.7 

60.6 


Teacher 

44.0 

58.1 

72.3 

75.6 

81.4 

82.6 


Student* 

75.8 

88.5 

90.9 

92.1 

93.2 

93.6 

High (%) 

Parent 

17.0 

23.6 

27.0 

33.3 

36.4 

35.0 


Teacher 

43.3 

60.1 

75.2 

80.7 

83.9 

85.0 


Student 

60.9 

71.1 

75.6 

77.2 

77.7 

76.4 


*Note: Only students in grades 6-12 respond to the survey. The response rate for elementary school students 
includes only the 6 ,h -graders in elementary schools. Similarly, response rates for K-8 schools only includes 6 th -8 th 
graders. 


Table 2: Number and percent of schools held accountable over time, by school type 



2007 

2008 

2009 

2010 

2011 

2012 

Elementary 

0 

570 

586 

587 

590 

594 


0% 

90% 

89% 

87% 

85% 

85% 

Middle 

0 

290 

323 

334 

352 

367 


0% 

82% 

89% 

88% 

90% 

95% 

K8 

0 

128 

140 

150 

155 

161 


0% 

90% 

99% 

99% 

100% 

100% 

High 

0 

197 

237 

248 

271 

280 


0% 

61% 

68% 

68% 

72% 

72% 


Note: schools are considered “accountable” if they have ever received a school letter grade. This table includes all 
schools, not just those in the analytic sample. 
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Table 3: NYC Learning Environment Survey sample domains and questions, 2007-2012 


Domain 

Teacher 

Parent 

Student 

Expectations 

■ My school sets high 
standards for student 
learning. 

■ My school sets high 
standards for student 
work in their classes. 

The school has high 
expectations for my 
child. 

■ I need to work hard 
to get good grades 
at my school. 

■ My teachers expect 
me to continue my 
education after high 
school. 

Discipline 

Order and discipline are 
maintained at my school. 

Discipline is enforced 
fairly at my child’s 
school. 

Discipline in my school 
is fair. 

School safety 

I am safe at my school. 

My child is safe at 
school. 

■ I am safe in my 
classes. 

■ I am safe in the 
hallways, 
bathrooms, and 
locker rooms. 

■ I am safe on school 
property outside 
my school building. 

Gang activity 

Gang activity is a 
problem in my school. 

There is gang activity in 
my child’s school. 

There is gang activity 
in my school. 

Bullying 

Students in my school are 
often threatened or 
bullied. 

Students threaten or bully 
other students. 

Students threaten or 
bully other students at 
school. 

Teachers’ respect 
for students 

Adults at my school are 
often disrespectful to 
students. 

School staff are 
disrespectful to students. 

Teachers in my school 
treat students with 
respect. 

Cleanliness 

The school is kept clean. 

My child’s school is 
clean 

My school is kept 
clean. 

Course / activity 
offerings 

My school offers a wide 
enough variety of 
activities or courses to 
keep students engaged at 
my school. 

My child's school offers a 
wide enough variety of 
courses and activities to 
keep my child interested 
in school 

My school offers a 
wide enough variety of 
classes and activities to 
keep me interested in 
school. 
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Table 4: Effect of accountability on within-group congruence (coefficient of variation), middle 
schools, 2007-2012 


Outcome: CV 

Expectations 

Parent Teacher Student 

(1) (2) (3) 

Parent 

(4) 

Discipline 

Teacher 

(5) 

Student 

(6) 

Accountable 

-0.025*** 

-0.010** 

0.028*** 

-0.017*** 

-0.028*** 

0.010*** 


(0.002) 

(0.004) 

(0.002) 

(0.003) 

(0.007) 

(0.003) 

Attendance rate 

-0.012** 

0.017 

0.012** 

0.003 

-0.011 

0.014* 


(0.006) 

(0.011) 

(0.005) 

(0.007) 

(0.017) 

(0.008) 

Suspension rate 

0.030** 

-0.005 

0.006 

0.069*** 

-0.001 

0.010 


(0.014) 

(0.024) 

(0.011) 

(0.016) 

(0.039) 

(0.017) 

Turnover rate 

0.020** 

0.015 

0.008 

0.008 

0.007 

0.004 


(0.008) 

(0.014) 

(0.006) 

(0.009) 

(0.022) 

(0.010) 

Enrollment (log) 

0.056*** 

0.038*** 

-0.003 

0.047*** 

0.066*** 

0.005 


(0.004) 

(0.007) 

(0.003) 

(0.005) 

(0.011) 

(0.005) 

Poor (share) 

-0.008 

-0.016* 

-0.011*** 

-0.006 

-0.009 

-0.017** 


(0.005) 

(0.009) 

(0.004) 

(0.006) 

(0.015) 

(0.006) 

LEP (share) 

-0.028 

-0.004 

0.018 

-0.037 

0.029 

0.005 


(0.034) 

(0.058) 

(0.028) 

(0.039) 

(0.095) 

(0.042) 

Black (share) 

0.045 

-0.038 

0.018 

0.088* 

0.236* 

0.089 


(0.044) 

(0.076) 

(0.036) 

(0.052) 

(0.125) 

(0.055) 

Hispanic (share) 

-0.015 

-0.047 

-0.003 

-0.013 

0.133 

-0.001 


(0.043) 

(0.075) 

(0.036) 

(0.051) 

(0.123) 

(0.054) 

Asian (share) 

0.061 

-0.009 

-0.020 

0.123** 

0.017 

-0.042 


(0.045) 

(0.078) 

(0.037) 

(0.053) 

(0.128) 

(0.056) 

Racial concentration 

-0.054** 

-0.038 

-0.023 

-0.023 

0.023 

-0.045 


(0.027) 

(0.047) 

(0.022) 

(0.032) 

(0.077) 

(0.034) 

Fewer 3 yrs exp (share) 0.000 

0.012 

0.038*** 

0.036*** 

0.005 

0.051*** 


(0.008) 

(0.014) 

(0.007) 

(0.010) 

(0.023) 

(0.010) 

Master’s + (share) 

0.001 

0.025 

0.007 

0.002 

-0.059* 

-0.010 


(0.012) 

(0.020) 

(0.010) 

(0.014) 

(0.033) 

(0.014) 

Not highly qual. (share) 0.028*** 

-0.020 

0.000 

0.022** 

-0.026 

-0.015 


(0.008) 

(0.014) 

(0.007) 

(0.009) 

(0.023) 

(0.010) 

Year trend 

Y 

Y 

Y 

Y 

Y 

Y 

School FE 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

2,021 

2,021 

2,021 

2,021 

2,021 

2,021 

R- squared 

0.706 

0.630 

0.608 

0.641 

0.666 

0.668 


Standard errors in parentheses 
*** p<0.01, ** p<0.05, * p<0.1 


Note: All rates and shares are measured from 0-1. Higher values of the coefficient of variation indicate less 
congruence; this means that negative coefficients indicate a variable is positively associated with congruence. Racial 
concentration is measured using a Herfindahl-Hirschman Index (HHI). Higher values of the HHI reflect greater 
racial concentration. If black, Hispanic, Asian, and white students were equally represented, the HHI would equal 
0.025; if only one student race group were represented in a school, the HHI would equal one. 
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Table 5: Regression results, effect of accountability on favorable individual perceptions of expectations and discipline (linear 
probability model); middle schools, 2007-2012 


VARIABLES 

(1) 

(2) 

Expectations 

(3) 


(4) 

(5) 

Discipline 

(6) 


Parent 

0.004*** 

-0.051*** 


-0.069*** 


0.184*** 

-0.323*** 


-0.310*** 


Student 

(0.001) 

0.041*** 

(0.017) 

-0.131*** 


(0.017) 

-0.103*** 


(0.002) 

-0.042*** 

(0.034) 

-0.342*** 


(0.035) 

-0.332*** 


Account*T 

(0.001) 

(0.016) 


(0.016) 

0.030*** 


(0.002) 

(0.033) 


(0.034) 

0.047*** 


Account*P 




(0.006) 

0.059*** 





(0.012) 

0.019*** 


Account*S 




(0.001) 

-0.020*** 





(0.002) 

0.028*** 






(0.001) 





(0.002) 


Schl controls 

N 

Y 


Y 


N 

Y 


Y 


Year trend 

Y 

Y 


Y 


Y 

Y 


Y 


School FE 

N 

Y 


Y 


N 

Y 


Y 


Between- group 

differences 


Pre-Acct 

Post-Acct 

Difference 



Pre-Acct 

Post-Acct 

Difference 

P-T Diff 

0.004*** 

-0.051*** 

-0.069*** 

-0.040*** 

0 . 029 *** 

0.184*** 

-0.323*** 

-0.310*** 

-0.338*** 

-0.028** 

S-T Diff 

0.041*** 

-0.131*** 

-0.103*** 

-0.153*** 

-0.050*** 

-0.042*** 

-0.342*** 

-0.332*** 

-0.351*** 

-0.019 

P-S Diff 

-0.037*** 

0.079*** 

0.034*** 

0.113*** 

0.079*** 

0.226*** 

0.019 

0.022 

0.013 

0 . 008 *** 

Observations 

1,479,690 

1,471,745 


1,471,745 


1,416,936 

1,409,316 


1,409,316 


R- squared 

0.010 

0.020 


0.023 


0.063 

0.096 


0.097 



Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Constant not shown. 


Note: Models include fully interacted control variables: attendance rate, suspension rate, % poor, LEP, black, Hispanic, Asian, racial concentration, fewer than 3 
yrs exp. Masters degrees, not highly qualified, turnover rate, and log of enrollment are all interacted with P and S (T is reference group, also included in model - 
not interacted). When the difference in pre- and post- accountability between-group gaps represents an increase in perceptual congruence (a narrowing of the 
gap), the value is bolded and underlined . The “accountable” variable takes a value of one if the school has ever received a school letter grade. 
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